2026-07-24 11:46:25,101 [ 384947 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:42, check_args_and_update_paths) 2026-07-24 11:46:25,101 [ 384947 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:86, check_args_and_update_paths) 2026-07-24 11:46:25,102 [ 384947 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:97, check_args_and_update_paths) 2026-07-24 11:46:25,102 [ 384947 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:99, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_cibjs0 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=b74d853ebe76 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_interserver_dns_retires/test.py::test_query test_jbod_ha/test.py::test_jbod_ha test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_availability_zone/test.py::test_get_availability_zone test_keeper_memory_soft_limit/test.py::test_soft_limit_create test_keeper_persistent_log_multinode/test.py::test_restart_multinode test_keeper_profiler/test.py::test_profiler test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader test_keeper_secure_client/test.py::test_connection test_library_bridge/test_exiled.py::test_bridge_dies_with_parent test_log_levels_update/test.py::test_log_levels_update test_max_authentication_methods_per_user/test.py::test_alter test_max_authentication_methods_per_user/test.py::test_alter_default_setting test_max_authentication_methods_per_user/test.py::test_create test_max_authentication_methods_per_user/test.py::test_create_default_setting test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 'test_merge_tree_s3/test.py::test_alter_table_columns[node]' 'test_merge_tree_s3/test.py::test_attach_detach_partition[node]' 'test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]' 'test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]' 'test_merge_tree_s3/test.py::test_freeze_unfreeze[node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]' 'test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]' 'test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]' 'test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]' 'test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]' 'test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]' 'test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node]' 'test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]' 'test_merge_tree_s3/test.py::test_table_manipulations[node]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]' test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1]' test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed test_placement_info/test.py::test_placement_info_from_config test_placement_info/test.py::test_placement_info_from_file test_placement_info/test.py::test_placement_info_from_imds test_placement_info/test.py::test_placement_info_missing_data test_postgresql_protocol/test.py::test_java_client test_postgresql_protocol/test.py::test_psql_client test_postgresql_protocol/test.py::test_python_client test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions test_postgresql_replica_database_engine_1/test.py::test_different_data_types test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries test_postgresql_replica_database_engine_1/test.py::test_multiple_databases test_postgresql_replica_database_engine_1/test.py::test_quoting_1 test_postgresql_replica_database_engine_1/test.py::test_quoting_2 test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index test_postgresql_replica_database_engine_1/test.py::test_replicating_dml test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished test_postgresql_replica_database_engine_1/test.py::test_single_transaction test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots test_postgresql_replica_database_engine_1/test.py::test_virtual_columns test_quota/test.py::test_add_remove_interval test_quota/test.py::test_add_remove_quota test_quota/test.py::test_consumption_of_show_clusters -vvv" altinityinfra/integration-tests-runner:cf95077f4609 '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: random-0.2, timeout-2.2.0, repeat-0.9.4, order-1.0.1, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication test_merge_tree_s3/test.py::test_alter_table_columns[node] test_max_authentication_methods_per_user/test.py::test_alter test_partition/test.py::test_attach_check_all_parts test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_postgresql_protocol/test.py::test_java_client test_placement_info/test.py::test_placement_info_from_config [gw4] [ 1%] PASSED test_max_authentication_methods_per_user/test.py::test_alter test_max_authentication_methods_per_user/test.py::test_alter_default_setting [gw4] [ 2%] PASSED test_max_authentication_methods_per_user/test.py::test_alter_default_setting test_max_authentication_methods_per_user/test.py::test_create [gw4] [ 3%] PASSED test_max_authentication_methods_per_user/test.py::test_create test_max_authentication_methods_per_user/test.py::test_create_default_setting [gw7] [ 4%] PASSED test_placement_info/test.py::test_placement_info_from_config test_placement_info/test.py::test_placement_info_from_file [gw4] [ 5%] PASSED test_max_authentication_methods_per_user/test.py::test_create_default_setting [gw3] [ 6%] PASSED test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part [gw3] [ 7%] PASSED test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists [gw3] [ 8%] PASSED test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts [gw7] [ 9%] PASSED test_placement_info/test.py::test_placement_info_from_file test_placement_info/test.py::test_placement_info_from_imds [gw3] [ 10%] PASSED test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached [gw7] [ 11%] PASSED test_placement_info/test.py::test_placement_info_from_imds test_placement_info/test.py::test_placement_info_missing_data [gw7] [ 12%] PASSED test_placement_info/test.py::test_placement_info_missing_data test_quota/test.py::test_add_remove_interval [gw5] [ 13%] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] [gw5] [ 14%] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] [gw9] [ 15%] PASSED test_postgresql_protocol/test.py::test_java_client test_postgresql_protocol/test.py::test_psql_client [gw6] [ 16%] PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where [gw3] [ 17%] PASSED test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex [gw2] [ 18%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] [gw5] [ 19%] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] [gw9] [ 20%] PASSED test_postgresql_protocol/test.py::test_psql_client test_postgresql_protocol/test.py::test_python_client [gw9] [ 21%] PASSED test_postgresql_protocol/test.py::test_python_client [gw6] [ 22%] PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where [gw3] [ 23%] PASSED test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple [gw5] [ 24%] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] [gw3] [ 25%] PASSED test_partition/test.py::test_partition_simple test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node test_partition/test.py::test_system_detached_parts [gw6] [ 26%] PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree [gw2] [ 27%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] [gw6] [ 28%] PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree [gw4] [ 29%] PASSED test_quota/test.py::test_add_remove_interval test_quota/test.py::test_add_remove_quota [gw0] [ 30%] PASSED test_merge_tree_s3/test.py::test_alter_table_columns[node] test_merge_tree_s3/test.py::test_attach_detach_partition[node] [gw2] [ 31%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] [gw0] [ 32%] PASSED test_merge_tree_s3/test.py::test_attach_detach_partition[node] test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] [gw4] [ 33%] PASSED test_quota/test.py::test_add_remove_quota test_quota/test.py::test_consumption_of_show_clusters [gw8] [ 34%] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted [gw4] [ 35%] PASSED test_quota/test.py::test_consumption_of_show_clusters [gw2] [ 36%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] test_keeper_memory_soft_limit/test.py::test_soft_limit_create [gw3] [ 37%] PASSED test_partition/test.py::test_system_detached_parts [gw2] [ 38%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] [gw7] [ 39%] PASSED test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed [gw7] [ 40%] PASSED test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed test_keeper_secure_client/test.py::test_connection [gw2] [ 41%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] test_keeper_persistent_log_multinode/test.py::test_restart_multinode [gw2] [ 42%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] [gw8] [ 43%] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error [gw2] [ 44%] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] [gw4] [ 45%] PASSED test_keeper_secure_client/test.py::test_connection test_jbod_ha/test.py::test_jbod_ha [gw8] [ 46%] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error test_keeper_profiler/test.py::test_profiler [gw0] [ 47%] PASSED test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] test_interserver_dns_retires/test.py::test_query [gw1] [ 48%] PASSED test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication [gw0] [ 49%] PASSED test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] test_merge_tree_s3/test.py::test_freeze_unfreeze[node] test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication test_library_bridge/test_exiled.py::test_bridge_dies_with_parent [gw0] [ 50%] PASSED test_merge_tree_s3/test.py::test_freeze_unfreeze[node] test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] test_log_levels_update/test.py::test_log_levels_update [gw0] [ 51%] PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] [gw0] [ 52%] PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] [gw8] [ 53%] PASSED test_log_levels_update/test.py::test_log_levels_update [gw0] [ 54%] PASSED test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] [gw5] [ 55%] PASSED test_interserver_dns_retires/test.py::test_query test_keeper_and_access_storage/test.py::test_create_replicated [gw3] [ 56%] PASSED test_keeper_persistent_log_multinode/test.py::test_restart_multinode test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 [gw0] [ 57%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] [gw6] [ 58%] PASSED test_keeper_memory_soft_limit/test.py::test_soft_limit_create [gw5] [ 59%] PASSED test_keeper_and_access_storage/test.py::test_create_replicated [gw7] [ 60%] PASSED test_keeper_profiler/test.py::test_profiler [gw0] [ 61%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] [gw4] [ 62%] PASSED test_library_bridge/test_exiled.py::test_bridge_dies_with_parent [gw0] [ 63%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] [gw3] [ 64%] PASSED test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 [gw0] [ 65%] PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] [gw9] [ 66%] PASSED test_jbod_ha/test.py::test_jbod_ha [gw0] [ 67%] PASSED test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader [gw1] [ 68%] PASSED test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication test_keeper_availability_zone/test.py::test_get_availability_zone [gw0] [ 69%] PASSED test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value [gw1] [ 70%] PASSED test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value [gw0] [ 71%] PASSED test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart [gw9] [ 72%] PASSED test_keeper_availability_zone/test.py::test_get_availability_zone [gw1] [ 73%] PASSED test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions [gw0] [ 74%] PASSED test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] [gw6] [ 75%] FAILED test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader [gw1] [ 76%] PASSED test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading test_postgresql_replica_database_engine_1/test.py::test_different_data_types [gw1] [ 77%] PASSED test_postgresql_replica_database_engine_1/test.py::test_different_data_types test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished [gw6] [ 78%] PASSED test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading [gw1] [ 79%] PASSED test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables [gw1] [ 80%] PASSED test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables [gw1] [ 81%] PASSED test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries [gw1] [ 82%] PASSED test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries test_postgresql_replica_database_engine_1/test.py::test_multiple_databases [gw1] [ 83%] PASSED test_postgresql_replica_database_engine_1/test.py::test_multiple_databases test_postgresql_replica_database_engine_1/test.py::test_quoting_1 [gw1] [ 84%] PASSED test_postgresql_replica_database_engine_1/test.py::test_quoting_1 test_postgresql_replica_database_engine_1/test.py::test_quoting_2 [gw1] [ 85%] PASSED test_postgresql_replica_database_engine_1/test.py::test_quoting_2 test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index [gw1] [ 86%] PASSED test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index test_postgresql_replica_database_engine_1/test.py::test_replicating_dml [gw1] [ 87%] PASSED test_postgresql_replica_database_engine_1/test.py::test_replicating_dml test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished [gw1] [ 88%] PASSED test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished test_postgresql_replica_database_engine_1/test.py::test_single_transaction [gw1] [ 89%] PASSED test_postgresql_replica_database_engine_1/test.py::test_single_transaction test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes [gw0] [ 90%] PASSED test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node] [gw0] [ 91%] SKIPPED test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node] test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node] [gw0] [ 92%] SKIPPED test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node] test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node] [gw0] [ 93%] SKIPPED test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node] test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] [gw0] [ 94%] PASSED test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] [gw1] [ 95%] PASSED test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots [gw0] [ 96%] PASSED test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] [gw0] [ 97%] PASSED test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] test_merge_tree_s3/test.py::test_table_manipulations[node] [gw0] [ 98%] PASSED test_merge_tree_s3/test.py::test_table_manipulations[node] [gw1] [ 99%] PASSED test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots test_postgresql_replica_database_engine_1/test.py::test_virtual_columns [gw1] [100%] PASSED test_postgresql_replica_database_engine_1/test.py::test_virtual_columns =================================== FAILURES =================================== ______________________ test_reconfig_remove_2_and_leader _______________________ [gw6] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_reconfig_remove_2_and_leader(started_cluster): """ Remove 2 followers from a cluster of 5. Remove leader from 3 nodes. """ global zk1, zk2, zk3, zk4, zk5 zk1 = create_client(node1) config = ku.get_config_str(zk1) print("Initial config", config) assert len(config.split("\n")) == 5 for i in range(100): zk1.create(f"/test_two_{i}", "somedata") zk4 = create_client(node4) zk4.sync("/test_two_0") ku.wait_configs_equal(config, zk4) zk5 = create_client(node5) zk5.sync("/test_two_0") ku.wait_configs_equal(config, zk5) for i in range(100): assert zk4.exists(f"test_two_{i}") assert zk5.exists(f"test_two_{i}") zk4.create(f"/test_two_{100 + i}", "otherdata") zk2 = create_client(node2) config = zk2.reconfig(joining=None, leaving="4,5", new_members=None) print("After removing 4,5", config) assert len(config.split("\n")) == 3 assert "node1" in config assert "node2" in config assert "node3" in config assert "node4" not in config assert "node5" not in config zk1.stop() zk1 = create_client(node1) zk1.sync("/test_two_0") ku.wait_configs_equal(config, zk1) for i in range(200): assert zk1.exists(f"test_two_{i}") assert zk2.exists(f"test_two_{i}") assert not node1.contains_in_log(log_msg_removed) assert not node2.contains_in_log(log_msg_removed) assert not node3.contains_in_log(log_msg_removed) assert node4.contains_in_log(log_msg_removed) assert node5.contains_in_log(log_msg_removed) assert ku.is_leader(cluster, node1) for i in range(100): zk1.create(f"/test_leader_{i}", "somedata") # when a leader gets a remove request, it must yield leadership config = zk1.reconfig(joining=None, leaving="1", new_members=None) print("After removing 1 (leader)", config) assert len(config.split("\n")) == 2 assert "node1" not in config assert "node2" in config assert "node3" in config assert "node4" not in config assert "node5" not in config zk2.stop() zk2 = create_client(node2) zk2.sync("/test_leader_0") ku.wait_configs_equal(config, zk2) zk3 = create_client(node3) zk3.sync("/test_leader_0") ku.wait_configs_equal(config, zk3) for i in range(100): assert zk2.exists(f"test_leader_{i}") > assert zk3.exists(f"test_leader_{i}") test_keeper_reconfig_remove_many/test.py:126: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/keeper_utils.py:147: in exists return bool(int(self.execute_query(f"exists '{path}'", timeout))) _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = query = "exists 'test_leader_93'", timeout = 60.0 def execute_query(self, query: str, timeout: float = 60.0) -> str: output = io.BytesIO() self.proc.stdin.write(query.encode() + b"\n") self.proc.stdin.flush() events = self.poller.poll(timeout) if not events: raise TimeoutError(f"Keeper client returned no output") for fd_num, event in events: if event & (select.EPOLLIN | select.EPOLLPRI): file = self._fd_nums[fd_num] if file == self.proc.stdout: while True: chunk = file.readline() if chunk.endswith(self.SEPARATOR): break output.write(chunk) elif file == self.proc.stderr: self.proc.stdout.readline() > raise KeeperException(self.proc.stderr.readline().strip().decode()) E helpers.keeper_utils.KeeperException: Code: 999. Coordination::Exception: Operation timeout (no response in 10000 ms) for request Exists for path: /test_leader_93. (KEEPER_EXCEPTION), Stack trace (when copying this message, always include the lines below): helpers/keeper_utils.py:118: KeeperException ---------------------------- Captured stdout setup ----------------------------- Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml Copy common default production configuration from /clickhouse-config. Files: config.xml, users.xml ------------------------------ Captured log setup ------------------------------ 2026-07-24 11:48:01 [ 532 ] INFO : Running tests in /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/test.py (cluster.py:2790, start) 2026-07-24 11:48:01 [ 532 ] DEBUG : Cluster start called. is_up=False (cluster.py:2797, start) 2026-07-24 11:48:01 [ 532 ] DEBUG : Docker networks for project roottestkeeperreconfigremovemany_gw6 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-07-24 11:48:01 [ 532 ] DEBUG : Docker containers for project roottestkeeperreconfigremovemany_gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-07-24 11:48:01 [ 532 ] DEBUG : Docker volumes for project roottestkeeperreconfigremovemany_gw6 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-07-24 11:48:01 [ 532 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-07-24 11:48:02 [ 532 ] DEBUG : Docker networks for project roottestkeeperreconfigremovemany_gw6 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-07-24 11:48:02 [ 532 ] DEBUG : Docker containers for project roottestkeeperreconfigremovemany_gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-07-24 11:48:02 [ 532 ] DEBUG : Docker volumes for project roottestkeeperreconfigremovemany_gw6 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-07-24 11:48:02 [ 532 ] DEBUG : Command:docker container list --all --filter name='^/roottestkeeperreconfigremovemany_gw6_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-07-24 11:48:02 [ 532 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-07-24 11:48:02 [ 532 ] DEBUG : No running containers for project: roottestkeeperreconfigremovemany_gw6 (cluster.py:904, cleanup) 2026-07-24 11:48:02 [ 532 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-07-24 11:48:02 [ 532 ] DEBUG : Trying to prune unused volumes... (cluster.py:937, cleanup) 2026-07-24 11:48:02 [ 532 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-07-24 11:48:02 [ 532 ] DEBUG : Stdout:6 (cluster.py:121, run_and_check) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup directory for instance: node1 (cluster.py:2810, start) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4538, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for common tests configuration (cluster.py:4543, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy common configuration from helpers (cluster.py:4563, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Generate and write macros file (cluster.py:4606, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper1.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/configs/config.d (cluster.py:4636, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/database (cluster.py:4653, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/logs (cluster.py:4664, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4750, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup directory for instance: node2 (cluster.py:2810, start) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4538, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for common tests configuration (cluster.py:4543, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy common configuration from helpers (cluster.py:4563, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Generate and write macros file (cluster.py:4606, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper2.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/configs/config.d (cluster.py:4636, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/database (cluster.py:4653, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/logs (cluster.py:4664, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4750, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup directory for instance: node3 (cluster.py:2810, start) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4538, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for common tests configuration (cluster.py:4543, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy common configuration from helpers (cluster.py:4563, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Generate and write macros file (cluster.py:4606, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper3.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/configs/config.d (cluster.py:4636, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/database (cluster.py:4653, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/logs (cluster.py:4664, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4750, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup directory for instance: node4 (cluster.py:2810, start) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4538, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for common tests configuration (cluster.py:4543, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy common configuration from helpers (cluster.py:4563, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Generate and write macros file (cluster.py:4606, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper4.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/configs/config.d (cluster.py:4636, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/database (cluster.py:4653, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/logs (cluster.py:4664, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4750, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup directory for instance: node5 (cluster.py:2810, start) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for configuration generated in this helper (cluster.py:4538, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Create directory for common tests configuration (cluster.py:4543, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy common configuration from helpers (cluster.py:4563, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Generate and write macros file (cluster.py:4606, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Copy custom test config files ['/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/configs/keeper5.xml'] to /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/configs/config.d (cluster.py:4636, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup database dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/database (cluster.py:4653, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Setup logs dir /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/logs (cluster.py:4664, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Entrypoint cmd: ["clickhouse", "server", "--config-file=/etc/clickhouse-server/config.xml", "--log-file=/var/log/clickhouse-server/clickhouse-server.log", "--errorlog-file=/var/log/clickhouse-server/clickhouse-server.err.log"] (cluster.py:4750, create_dir) 2026-07-24 11:48:02 [ 532 ] DEBUG : Env {'ASAN_OPTIONS': 'use_sigaltstack=0', 'TSAN_OPTIONS': 'use_sigaltstack=0', 'CLICKHOUSE_WATCHDOG_ENABLE': '0', 'CLICKHOUSE_NATS_TLS_SECURE': '0', 'LLVM_PROFILE_FILE': '/var/lib/clickhouse/server_%h_%p_%m.profraw'} stored in /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/.env (cluster.py:86, _create_env_file) 2026-07-24 11:48:02 [ 532 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-07-24 11:48:02 [ 532 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-07-24 11:48:02 [ 532 ] DEBUG : Trying paths: ['/root/.docker/config.json', '/root/.dockercfg'] (config.py:21, find_config_file) 2026-07-24 11:48:02 [ 532 ] DEBUG : No config file found (config.py:28, find_config_file) 2026-07-24 11:48:02 [ 532 ] DEBUG : http://localhost:None "GET /version HTTP/1.1" 200 825 (connectionpool.py:547, _make_request) 2026-07-24 11:48:02 [ 532 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/.env', '--project-name', 'roottestkeeperreconfigremovemany_gw6', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/docker-compose.yml', 'pull'] (cluster.py:113, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node3 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node4 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node5 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node2 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node5 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node3 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node4 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node2 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node1 ... pulling from altinityinfra/integr... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node3 ... digest: sha256:2ea9da862c215fd684... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node3 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node4 ... digest: sha256:2ea9da862c215fd684... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node4 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node3 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node5 ... digest: sha256:2ea9da862c215fd684... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node5 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node4 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node2 ... digest: sha256:2ea9da862c215fd684... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node2 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node1 ... digest: sha256:2ea9da862c215fd684... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node1 ... status: image is up to date for a... (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node5 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node2 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : Stderr:Pulling node1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:04 [ 532 ] DEBUG : ('Trying to create ClickHouse instance by command %s', 'docker-compose --env-file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/.env --project-name roottestkeeperreconfigremovemany_gw6 --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/docker-compose.yml --file /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/docker-compose.yml up -d --no-recreate') (cluster.py:3150, start) 2026-07-24 11:48:04 [ 532 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/.env', '--project-name', 'roottestkeeperreconfigremovemany_gw6', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/docker-compose.yml', 'up', '-d', '--no-recreate'] (cluster.py:113, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating network "roottestkeeperreconfigremovemany_gw6_default" with the default driver (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node5_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node3_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node4_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node2_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node1_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node5_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node2_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node1_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node4_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : Stderr:Creating roottestkeeperreconfigremovemany_gw6_node3_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:07 [ 532 ] DEBUG : ClickHouse instance created (cluster.py:3158, start) 2026-07-24 11:48:07 [ 532 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:07 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:07 [ 532 ] DEBUG : Waiting for ClickHouse start in node1, ip: 172.16.23.4... (cluster.py:3165, start) 2026-07-24 11:48:07 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:07 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:07 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:07 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:07 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:08 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/594917b5975e2374516d56553df78e2657b1ea730c8f134aa4740ec9570b8145/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : ClickHouse node1 started (cluster.py:3169, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : Waiting for ClickHouse start in node2, ip: 172.16.23.3... (cluster.py:3165, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/b35dc0d0eb7a69cd30f7ca2ea343ba91ec3995ce2dd99e3d859c350c0d9b26e1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : ClickHouse node2 started (cluster.py:3169, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : Waiting for ClickHouse start in node3, ip: 172.16.23.5... (cluster.py:3165, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/e9e2640b247a98b93a21004e327c474b27b05630fcc0d4823947ffbf7d94704d/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : ClickHouse node3 started (cluster.py:3169, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node4_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : Waiting for ClickHouse start in node4, ip: 172.16.23.6... (cluster.py:3165, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node4_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/88d318fef867a4defd998f1ebb0b3a6d6889e11645dd6f22a8e5922adc297968/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : ClickHouse node4 started (cluster.py:3169, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node5_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : Waiting for ClickHouse start in node5, ip: 172.16.23.2... (cluster.py:3165, start) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node5_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/a8369985afe3360f6d0d3b26bc05a068a14630d079978dc107ec52ddda5cc7bb/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:09 [ 532 ] DEBUG : ClickHouse node5 started (cluster.py:3169, start) ----------------------------- Captured stdout call ----------------------------- Initial config server.1=node1:9234;participant;1 server.2=node2:9234;participant;1 server.3=node3:9234;participant;1 server.4=node4:9234;participant;1 server.5=node5:9234;participant;1 After removing 4,5 server.1=node1:9234;participant;1 server.2=node2:9234;participant;1 server.3=node3:9234;participant;1 After removing 1 (leader) server.2=node2:9234;participant;1 server.3=node3:9234;participant;1 ------------------------------ Captured log call ------------------------------- 2026-07-24 11:48:09 [ 532 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:09 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:10 [ 532 ] DEBUG : get_instance_ip instance_name=node4 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:10 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node4_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:10 [ 532 ] DEBUG : get_instance_ip instance_name=node5 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:10 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node5_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:12 [ 532 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:12 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:12 [ 532 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:12 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:13 [ 532 ] DEBUG : run container_id:roottestkeeperreconfigremovemany_gw6_node1_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2175, exec_in_container) 2026-07-24 11:48:13 [ 532 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperreconfigremovemany_gw6_node1_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:13 [ 532 ] DEBUG : run container_id:roottestkeeperreconfigremovemany_gw6_node2_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2175, exec_in_container) 2026-07-24 11:48:13 [ 532 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperreconfigremovemany_gw6_node2_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:13 [ 532 ] DEBUG : run container_id:roottestkeeperreconfigremovemany_gw6_node3_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2175, exec_in_container) 2026-07-24 11:48:13 [ 532 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperreconfigremovemany_gw6_node3_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:13 [ 532 ] DEBUG : run container_id:roottestkeeperreconfigremovemany_gw6_node4_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2175, exec_in_container) 2026-07-24 11:48:13 [ 532 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperreconfigremovemany_gw6_node4_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:14 [ 532 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2026.07.24 11:48:12.385451 [ 604 ] {} RaftInstance: this server (4) has been removed from the cluster, will step down itself soon. config log idx 239 (cluster.py:121, run_and_check) 2026-07-24 11:48:14 [ 532 ] DEBUG : run container_id:roottestkeeperreconfigremovemany_gw6_node5_1 detach:False nothrow:False cmd: ['bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:2175, exec_in_container) 2026-07-24 11:48:14 [ 532 ] DEBUG : Command:['docker', 'exec', 'roottestkeeperreconfigremovemany_gw6_node5_1', 'bash', '-c', '[ -f /var/log/clickhouse-server/clickhouse-server.log ] && zgrep -aH "has been removed from the cluster" /var/log/clickhouse-server/clickhouse-server.log | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:14 [ 532 ] DEBUG : Stdout:/var/log/clickhouse-server/clickhouse-server.log:2026.07.24 11:48:12.434938 [ 604 ] {} RaftInstance: this server (5) has been removed from the cluster, will step down itself soon. config log idx 243 (cluster.py:121, run_and_check) 2026-07-24 11:48:14 [ 532 ] DEBUG : get_instance_ip instance_name=node1 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:14 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node1_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:15 [ 532 ] DEBUG : get_instance_ip instance_name=node2 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:15 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node2_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) 2026-07-24 11:48:29 [ 532 ] WARNING : Connection dropped: socket connection error: None (connection.py:622, _connect_attempt) 2026-07-24 11:48:29 [ 532 ] INFO : Connecting to 172.16.13.2(172.16.13.2):2181, use_ssl: False (connection.py:650, _connect) 2026-07-24 11:48:31 [ 532 ] DEBUG : get_instance_ip instance_name=node3 (cluster.py:2137, get_instance_ip) 2026-07-24 11:48:31 [ 532 ] DEBUG : http://localhost:None "GET /v1.42/containers/roottestkeeperreconfigremovemany_gw6_node3_1/json HTTP/1.1" 200 None (connectionpool.py:547, _make_request) ---------------------------- Captured log teardown ----------------------------- 2026-07-24 11:48:42 [ 532 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/.env', '--project-name', 'roottestkeeperreconfigremovemany_gw6', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/docker-compose.yml', 'stop', '--timeout', '20'] (cluster.py:113, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node3_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node2_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node1_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node5_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node4_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node5_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node4_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node3_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node2_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Stderr:Stopping roottestkeeperreconfigremovemany_gw6_node1_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Command:['bash', '-c', '[ -f /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true'] (cluster.py:113, run_and_check) 2026-07-24 11:48:50 [ 532 ] DEBUG : Command:['docker-compose', '--env-file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/.env', '--project-name', 'roottestkeeperreconfigremovemany_gw6', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node1/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node2/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node3/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node4/docker-compose.yml', '--file', '/ClickHouse/tests/integration/test_keeper_reconfig_remove_many/_instances_0_gw6/node5/docker-compose.yml', 'down', '--volumes'] (cluster.py:113, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node3_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node2_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node1_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node5_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node4_1 ... (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node2_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node1_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node4_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node5_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing roottestkeeperreconfigremovemany_gw6_node3_1 ... done (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stderr:Removing network roottestkeeperreconfigremovemany_gw6_default (cluster.py:123, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Cleanup called (cluster.py:876, cleanup) 2026-07-24 11:48:51 [ 532 ] DEBUG : Docker networks for project roottestkeeperreconfigremovemany_gw6 are NETWORK ID NAME DRIVER SCOPE (cluster.py:855, print_all_docker_pieces) 2026-07-24 11:48:51 [ 532 ] DEBUG : Docker containers for project roottestkeeperreconfigremovemany_gw6 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:863, print_all_docker_pieces) 2026-07-24 11:48:51 [ 532 ] DEBUG : Docker volumes for project roottestkeeperreconfigremovemany_gw6 are DRIVER VOLUME NAME (cluster.py:871, print_all_docker_pieces) 2026-07-24 11:48:51 [ 532 ] DEBUG : Command:docker container list --all --filter name='^/roottestkeeperreconfigremovemany_gw6_.*_1$' --format '{{.ID}}:{{.Names}}' (cluster.py:113, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Unstopped containers: {} (cluster.py:890, cleanup) 2026-07-24 11:48:51 [ 532 ] DEBUG : No running containers for project: roottestkeeperreconfigremovemany_gw6 (cluster.py:904, cleanup) 2026-07-24 11:48:51 [ 532 ] DEBUG : Trying to prune unused networks... (cluster.py:910, cleanup) 2026-07-24 11:48:51 [ 532 ] DEBUG : Trying to prune unused volumes... (cluster.py:937, cleanup) 2026-07-24 11:48:51 [ 532 ] DEBUG : Command:['docker volume ls | wc -l'] (cluster.py:113, run_and_check) 2026-07-24 11:48:51 [ 532 ] DEBUG : Stdout:4 (cluster.py:121, run_and_check) =============================== warnings summary =============================== test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries /usr/local/lib/python3.10/dist-packages/_pytest/threadexception.py:73: PytestUnhandledThreadExceptionWarning: Exception in thread Thread-35 (attack) Traceback (most recent call last): File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner self.run() File "/usr/lib/python3.10/threading.py", line 953, in run self._target(*self._args, **self._kwargs) File "/ClickHouse/tests/integration/test_postgresql_replica_database_engine_1/test.py", line 431, in attack cursor.execute(query_pool[query_id].format(random_table_name)) psycopg2.errors.NumericValueOutOfRange: integer out of range warnings.warn(pytest.PytestUnhandledThreadExceptionWarning(msg)) -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html ============================== slowest durations =============================== 124.47s call test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] 38.90s call test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication 34.53s call test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication 31.88s call test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader 31.47s call test_keeper_memory_soft_limit/test.py::test_soft_limit_create 26.72s call test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions 25.97s teardown test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 25.13s call test_jbod_ha/test.py::test_jbod_ha 24.86s teardown test_postgresql_protocol/test.py::test_python_client 24.06s teardown test_merge_tree_s3/test.py::test_table_manipulations[node] 23.96s teardown test_keeper_profiler/test.py::test_profiler 22.10s setup test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams 21.57s setup test_postgresql_protocol/test.py::test_java_client 20.78s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] 20.10s setup test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 19.92s setup test_merge_tree_s3/test.py::test_alter_table_columns[node] 19.90s call test_keeper_persistent_log_multinode/test.py::test_restart_multinode 19.45s setup test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication 19.03s setup test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 18.73s call test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 18.21s call test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] 17.80s call test_interserver_dns_retires/test.py::test_query 17.21s call test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] 15.36s teardown test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 15.30s call test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries 14.24s setup test_placement_info/test.py::test_placement_info_from_config 14.16s call test_keeper_profiler/test.py::test_profiler 13.92s setup test_keeper_profiler/test.py::test_profiler 13.90s call test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 13.64s setup test_jbod_ha/test.py::test_jbod_ha 13.36s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] 13.06s setup test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading 12.86s call test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished 12.04s setup test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node 11.80s teardown test_keeper_memory_soft_limit/test.py::test_soft_limit_create 11.77s call test_postgresql_replica_database_engine_1/test.py::test_single_transaction 11.48s teardown test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed 11.18s call test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart 11.09s call test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 11.01s setup test_keeper_persistent_log_multinode/test.py::test_restart_multinode 10.91s teardown test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading 10.91s setup test_partition/test.py::test_attach_check_all_parts 10.72s setup test_keeper_memory_soft_limit/test.py::test_soft_limit_create 10.50s call test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] 10.23s setup test_keeper_availability_zone/test.py::test_get_availability_zone 10.23s setup test_max_authentication_methods_per_user/test.py::test_alter 9.98s teardown test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader 9.61s call test_merge_tree_s3/test.py::test_alter_table_columns[node] 9.21s call test_postgresql_replica_database_engine_1/test.py::test_multiple_databases 8.84s call test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots 8.82s call test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished 8.14s setup test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 8.09s call test_partition/test.py::test_system_detached_parts 8.02s call test_postgresql_replica_database_engine_1/test.py::test_replicating_dml 8.00s call test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables 7.87s setup test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader 7.56s setup test_keeper_secure_client/test.py::test_connection 7.34s setup test_keeper_and_access_storage/test.py::test_create_replicated 7.24s teardown test_keeper_secure_client/test.py::test_connection 7.17s setup test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 6.58s call test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] 6.26s teardown test_placement_info/test.py::test_placement_info_missing_data 6.18s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 6.09s teardown test_quota/test.py::test_consumption_of_show_clusters 6.03s setup test_quota/test.py::test_add_remove_interval 6.01s teardown test_log_levels_update/test.py::test_log_levels_update 5.93s teardown test_partition/test.py::test_system_detached_parts 5.90s teardown test_max_authentication_methods_per_user/test.py::test_create_default_setting 5.69s teardown test_jbod_ha/test.py::test_jbod_ha 5.61s teardown test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree 5.23s call test_postgresql_replica_database_engine_1/test.py::test_different_data_types 5.22s call test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 5.21s setup test_log_levels_update/test.py::test_log_levels_update 5.15s call test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading 5.12s call test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes 4.77s teardown test_keeper_availability_zone/test.py::test_get_availability_zone 4.66s teardown test_postgresql_replica_database_engine_1/test.py::test_virtual_columns 4.47s call test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] 4.26s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] 4.12s teardown test_keeper_persistent_log_multinode/test.py::test_restart_multinode 4.00s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] 3.97s call test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] 3.97s call test_partition/test.py::test_make_clone_in_detached 3.88s call test_partition/test.py::test_attach_check_all_parts 3.74s call test_merge_tree_s3/test.py::test_attach_detach_partition[node] 3.69s call test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 3.63s call test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] 3.57s teardown test_keeper_and_access_storage/test.py::test_create_replicated 3.50s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] 3.48s call test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] 3.47s call test_quota/test.py::test_add_remove_quota 3.43s call test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables 3.25s teardown test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions 3.21s teardown test_library_bridge/test_exiled.py::test_bridge_dies_with_parent 3.17s call test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] 3.17s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] 3.16s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] 3.10s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] 3.07s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] 2.99s call test_merge_tree_s3/test.py::test_table_manipulations[node] 2.82s call test_placement_info/test.py::test_placement_info_from_file 2.78s call test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index 2.72s call test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] 2.72s call test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] 2.71s call test_quota/test.py::test_add_remove_interval 2.70s call test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] 2.70s call test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] 2.69s teardown test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication 2.55s call test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] 2.51s call test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] 2.51s call test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree 2.44s call test_max_authentication_methods_per_user/test.py::test_alter 2.41s call test_placement_info/test.py::test_placement_info_from_imds 2.38s call test_log_levels_update/test.py::test_log_levels_update 2.33s call test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value 2.17s call test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 1.78s call test_partition/test.py::test_detached_part_dir_exists 1.77s call test_merge_tree_s3/test.py::test_freeze_unfreeze[node] 1.68s call test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 1.65s call test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where 1.64s call test_max_authentication_methods_per_user/test.py::test_alter_default_setting 1.59s call test_postgresql_protocol/test.py::test_java_client 1.59s call test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where 1.57s call test_partition/test.py::test_drop_detached_parts 1.48s call test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams 1.44s call test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 1.41s call test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 1.39s call test_postgresql_protocol/test.py::test_psql_client 1.32s teardown test_postgresql_replica_database_engine_1/test.py::test_multiple_databases 1.31s call test_partition/test.py::test_partition_complex 1.23s call test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] 1.17s call test_postgresql_replica_database_engine_1/test.py::test_virtual_columns 1.03s teardown test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication 0.93s call test_postgresql_replica_database_engine_1/test.py::test_quoting_1 0.89s call test_postgresql_replica_database_engine_1/test.py::test_quoting_2 0.79s teardown test_postgresql_replica_database_engine_1/test.py::test_single_transaction 0.72s teardown test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables 0.67s teardown test_postgresql_replica_database_engine_1/test.py::test_quoting_2 0.66s teardown test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes 0.65s teardown test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index 0.65s teardown test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished 0.63s teardown test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries 0.62s teardown test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables 0.61s call test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node 0.61s teardown test_postgresql_replica_database_engine_1/test.py::test_replicating_dml 0.60s teardown test_postgresql_replica_database_engine_1/test.py::test_quoting_1 0.56s setup test_quota/test.py::test_add_remove_quota 0.55s call test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed 0.55s setup test_partition/test.py::test_system_detached_parts 0.54s setup test_quota/test.py::test_consumption_of_show_clusters 0.52s teardown test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished 0.52s teardown test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value 0.50s teardown test_postgresql_replica_database_engine_1/test.py::test_different_data_types 0.49s teardown test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots 0.49s setup test_partition/test.py::test_drop_detached_parts 0.47s setup test_partition/test.py::test_partition_complex 0.47s setup test_partition/test.py::test_partition_simple 0.46s teardown test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart 0.45s setup test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] 0.45s call test_placement_info/test.py::test_placement_info_from_config 0.42s call test_keeper_and_access_storage/test.py::test_create_replicated 0.35s call test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] 0.35s teardown test_partition/test.py::test_drop_detached_parts 0.33s call test_placement_info/test.py::test_placement_info_missing_data 0.32s call test_partition/test.py::test_cannot_attach_active_part 0.27s call test_partition/test.py::test_partition_simple 0.23s setup test_partition/test.py::test_cannot_attach_active_part 0.21s teardown test_partition/test.py::test_attach_check_all_parts 0.20s call test_max_authentication_methods_per_user/test.py::test_create_default_setting 0.19s call test_quota/test.py::test_consumption_of_show_clusters 0.18s call test_max_authentication_methods_per_user/test.py::test_create 0.15s call test_keeper_availability_zone/test.py::test_get_availability_zone 0.14s setup test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] 0.13s call test_postgresql_protocol/test.py::test_python_client 0.13s setup test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] 0.12s setup test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node] 0.12s setup test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node] 0.12s setup test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node] 0.12s teardown test_partition/test.py::test_cannot_attach_active_part 0.12s teardown test_partition/test.py::test_partition_complex 0.12s call test_keeper_secure_client/test.py::test_connection 0.08s setup test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] 0.07s teardown test_partition/test.py::test_partition_simple 0.02s setup test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] 0.01s setup test_merge_tree_s3/test.py::test_attach_detach_partition[node] 0.01s setup test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] 0.01s setup test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] 0.01s setup test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] 0.01s setup test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] 0.01s setup test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] 0.01s setup test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] 0.01s setup test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] 0.01s setup test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] 0.01s setup test_merge_tree_s3/test.py::test_table_manipulations[node] 0.00s setup test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] 0.00s setup test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] 0.00s setup test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] 0.00s teardown test_postgresql_protocol/test.py::test_java_client 0.00s setup test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] 0.00s setup test_merge_tree_s3/test.py::test_freeze_unfreeze[node] 0.00s teardown test_merge_tree_s3/test.py::test_alter_table_columns[node] 0.00s teardown test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] 0.00s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts 0.00s teardown test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] 0.00s setup test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] 0.00s teardown test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_multiple_databases 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_virtual_columns 0.00s setup test_max_authentication_methods_per_user/test.py::test_create 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_quoting_2 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_replicating_dml 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_single_transaction 0.00s teardown test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_quoting_1 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_different_data_types 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished 0.00s setup test_interserver_dns_retires/test.py::test_query 0.00s setup test_placement_info/test.py::test_placement_info_from_imds 0.00s teardown test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication 0.00s setup test_max_authentication_methods_per_user/test.py::test_alter_default_setting 0.00s setup test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed 0.00s teardown test_interserver_dns_retires/test.py::test_query 0.00s teardown test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] 0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] 0.00s setup test_placement_info/test.py::test_placement_info_missing_data 0.00s setup test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions 0.00s teardown test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] 0.00s teardown test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] 0.00s call test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node] 0.00s setup test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] 0.00s setup test_postgresql_protocol/test.py::test_psql_client 0.00s call test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node] 0.00s setup test_postgresql_protocol/test.py::test_python_client 0.00s setup test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 0.00s teardown test_partition/test.py::test_make_clone_in_detached 0.00s call test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node] 0.00s teardown test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] 0.00s teardown test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node] 0.00s setup test_placement_info/test.py::test_placement_info_from_file 0.00s teardown test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] 0.00s teardown test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node] 0.00s setup test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where 0.00s teardown test_placement_info/test.py::test_placement_info_from_config 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] 0.00s teardown test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] 0.00s teardown test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] 0.00s teardown test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] 0.00s teardown test_max_authentication_methods_per_user/test.py::test_alter 0.00s teardown test_postgresql_protocol/test.py::test_psql_client 0.00s teardown test_merge_tree_s3/test.py::test_attach_detach_partition[node] 0.00s setup test_partition/test.py::test_detached_part_dir_exists 0.00s teardown test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 0.00s setup test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted 0.00s setup test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] 0.00s teardown test_partition/test.py::test_detached_part_dir_exists 0.00s teardown test_max_authentication_methods_per_user/test.py::test_alter_default_setting 0.00s setup test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] 0.00s teardown test_quota/test.py::test_add_remove_interval 0.00s setup test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where 0.00s setup test_max_authentication_methods_per_user/test.py::test_create_default_setting 0.00s teardown test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] 0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] 0.00s setup test_partition/test.py::test_make_clone_in_detached 0.00s teardown test_merge_tree_s3/test.py::test_freeze_unfreeze[node] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] 0.00s setup test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree 0.00s teardown test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] 0.00s teardown test_quota/test.py::test_add_remove_quota 0.00s teardown test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] 0.00s teardown test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] 0.00s teardown test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] 0.00s teardown test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] 0.00s teardown test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] 0.00s teardown test_placement_info/test.py::test_placement_info_from_file 0.00s teardown test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] 0.00s teardown test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] 0.00s teardown test_placement_info/test.py::test_placement_info_from_imds 0.00s teardown test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where 0.00s teardown test_max_authentication_methods_per_user/test.py::test_create =========================== short test summary info ============================ FAILED test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader PASSED test_max_authentication_methods_per_user/test.py::test_alter PASSED test_max_authentication_methods_per_user/test.py::test_alter_default_setting PASSED test_max_authentication_methods_per_user/test.py::test_create PASSED test_placement_info/test.py::test_placement_info_from_config PASSED test_max_authentication_methods_per_user/test.py::test_create_default_setting PASSED test_partition/test.py::test_attach_check_all_parts PASSED test_partition/test.py::test_cannot_attach_active_part PASSED test_partition/test.py::test_detached_part_dir_exists PASSED test_placement_info/test.py::test_placement_info_from_file PASSED test_partition/test.py::test_drop_detached_parts PASSED test_placement_info/test.py::test_placement_info_from_imds PASSED test_placement_info/test.py::test_placement_info_missing_data PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2] PASSED test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1] PASSED test_postgresql_protocol/test.py::test_java_client PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams PASSED test_partition/test.py::test_make_clone_in_detached PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0] PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2] PASSED test_postgresql_protocol/test.py::test_psql_client PASSED test_postgresql_protocol/test.py::test_python_client PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where PASSED test_partition/test.py::test_partition_complex PASSED test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1] PASSED test_partition/test.py::test_partition_simple PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1] PASSED test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree PASSED test_quota/test.py::test_add_remove_interval PASSED test_merge_tree_s3/test.py::test_alter_table_columns[node] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0] PASSED test_merge_tree_s3/test.py::test_attach_detach_partition[node] PASSED test_quota/test.py::test_add_remove_quota PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts PASSED test_quota/test.py::test_consumption_of_show_clusters PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1] PASSED test_partition/test.py::test_system_detached_parts PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0] PASSED test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node PASSED test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1] PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0] PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted PASSED test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1] PASSED test_keeper_secure_client/test.py::test_connection PASSED test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error PASSED test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk] PASSED test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication PASSED test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node] PASSED test_merge_tree_s3/test.py::test_freeze_unfreeze[node] PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node] PASSED test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node] PASSED test_log_levels_update/test.py::test_log_levels_update PASSED test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node] PASSED test_interserver_dns_retires/test.py::test_query PASSED test_keeper_persistent_log_multinode/test.py::test_restart_multinode PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node] PASSED test_keeper_memory_soft_limit/test.py::test_soft_limit_create PASSED test_keeper_and_access_storage/test.py::test_create_replicated PASSED test_keeper_profiler/test.py::test_profiler PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3] PASSED test_library_bridge/test_exiled.py::test_bridge_dies_with_parent PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part] PASSED test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 PASSED test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node] PASSED test_jbod_ha/test.py::test_jbod_ha PASSED test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node] PASSED test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication PASSED test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node] PASSED test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value PASSED test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node] PASSED test_keeper_availability_zone/test.py::test_get_availability_zone PASSED test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart PASSED test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node] PASSED test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions PASSED test_postgresql_replica_database_engine_1/test.py::test_different_data_types PASSED test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading PASSED test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished PASSED test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables PASSED test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables PASSED test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries PASSED test_postgresql_replica_database_engine_1/test.py::test_multiple_databases PASSED test_postgresql_replica_database_engine_1/test.py::test_quoting_1 PASSED test_postgresql_replica_database_engine_1/test.py::test_quoting_2 PASSED test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index PASSED test_postgresql_replica_database_engine_1/test.py::test_replicating_dml PASSED test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished PASSED test_postgresql_replica_database_engine_1/test.py::test_single_transaction PASSED test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node] PASSED test_merge_tree_s3/test.py::test_s3_no_delete_objects[node] PASSED test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes PASSED test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node] PASSED test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node] PASSED test_merge_tree_s3/test.py::test_table_manipulations[node] PASSED test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots PASSED test_postgresql_replica_database_engine_1/test.py::test_virtual_columns SKIPPED [3] test_merge_tree_s3/test.py:931: Disabled, will be fixed after https://github.com/ClickHouse/ClickHouse/issues/51152 ======== 1 failed, 96 passed, 3 skipped, 1 warning in 282.86s (0:04:42) ======== Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 445, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_cibjs0 --privileged --dns-search='.' --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=2cffe1eae894 -e DOCKER_BASE_TAG=b74d853ebe76 -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 test_interserver_dns_retires/test.py::test_query test_jbod_ha/test.py::test_jbod_ha test_keeper_and_access_storage/test.py::test_create_replicated test_keeper_availability_zone/test.py::test_get_availability_zone test_keeper_memory_soft_limit/test.py::test_soft_limit_create test_keeper_persistent_log_multinode/test.py::test_restart_multinode test_keeper_profiler/test.py::test_profiler test_keeper_reconfig_remove/test.py::test_reconfig_remove_followers_from_3 test_keeper_reconfig_remove_many/test.py::test_reconfig_remove_2_and_leader test_keeper_secure_client/test.py::test_connection test_library_bridge/test_exiled.py::test_bridge_dies_with_parent test_log_levels_update/test.py::test_log_levels_update test_max_authentication_methods_per_user/test.py::test_alter test_max_authentication_methods_per_user/test.py::test_alter_default_setting test_max_authentication_methods_per_user/test.py::test_create test_max_authentication_methods_per_user/test.py::test_create_default_setting test_merge_tree_load_parts/test.py::test_merge_tree_load_parts test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_corrupted test_merge_tree_load_parts/test.py::test_merge_tree_load_parts_filesystem_error 'test_merge_tree_s3/test.py::test_alter_table_columns[node]' 'test_merge_tree_s3/test.py::test_attach_detach_partition[node]' 'test_merge_tree_s3/test.py::test_cache_with_full_disk_space[node_with_limited_disk]' 'test_merge_tree_s3/test.py::test_freeze_system_unfreeze[node]' 'test_merge_tree_s3/test.py::test_freeze_unfreeze[node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[False-node]' 'test_merge_tree_s3/test.py::test_insert_same_partition_and_merge[True-node]' 'test_merge_tree_s3/test.py::test_lazy_seek_optimization_for_async_read[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_drop[node]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors[node-broken_s3_always_multi_part]' 'test_merge_tree_s3/test.py::test_merge_canceled_by_s3_errors_when_move[node]' 'test_merge_tree_s3/test.py::test_move_partition_to_another_disk[node]' 'test_merge_tree_s3/test.py::test_move_replace_partition_to_another_table[node]' 'test_merge_tree_s3/test.py::test_s3_disk_apply_new_settings[node]' 'test_merge_tree_s3/test.py::test_s3_disk_heavy_write_check_mem[node]' 'test_merge_tree_s3/test.py::test_s3_disk_reads_on_unstable_connection[node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory0-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory1-node]' 'test_merge_tree_s3/test.py::test_s3_engine_heavy_write_check_mem[in_flight_memory2-node]' 'test_merge_tree_s3/test.py::test_s3_no_delete_objects[node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[0-16-node]' 'test_merge_tree_s3/test.py::test_simple_insert_select[8192-12-node]' 'test_merge_tree_s3/test.py::test_table_manipulations[node]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[0-4-2]' 'test_merge_tree_s3_with_cache/test.py::test_read_after_cache_is_wiped[8192-2-1]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[0-2]' 'test_merge_tree_s3_with_cache/test.py::test_write_is_cached[8192-1]' test_mutations_in_partitions_of_merge_tree/test.py::test_mutation_max_streams test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_with_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_merge_tree_without_where test_mutations_in_partitions_of_merge_tree/test.py::test_trivial_alter_in_partition_replicated_merge_tree 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_range-sipHash64(key)-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-key-1]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-0]' 'test_parallel_replicas_custom_key_load_balancing/test.py::test_parallel_replicas_custom_key_load_balancing[custom_key_sampling-sipHash64(key)-1]' test_parallel_replicas_snapshot_from_initiator/test.py::test_initiator_snapshot_is_used_for_reading test_partition/test.py::test_attach_check_all_parts test_partition/test.py::test_cannot_attach_active_part test_partition/test.py::test_detached_part_dir_exists test_partition/test.py::test_drop_detached_parts test_partition/test.py::test_make_clone_in_detached test_partition/test.py::test_partition_complex test_partition/test.py::test_partition_simple test_partition/test.py::test_system_detached_parts test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_single_node test_peak_memory_usage/test.py::test_clickhouse_client_max_peak_memory_usage_distributed test_placement_info/test.py::test_placement_info_from_config test_placement_info/test.py::test_placement_info_from_file test_placement_info/test.py::test_placement_info_from_imds test_placement_info/test.py::test_placement_info_missing_data test_postgresql_protocol/test.py::test_java_client test_postgresql_protocol/test.py::test_psql_client test_postgresql_protocol/test.py::test_python_client test_postgresql_replica_database_engine_1/test.py::test_abrupt_connection_loss_while_heavy_replication test_postgresql_replica_database_engine_1/test.py::test_abrupt_server_restart_while_heavy_replication test_postgresql_replica_database_engine_1/test.py::test_changing_replica_identity_value test_postgresql_replica_database_engine_1/test.py::test_clickhouse_restart test_postgresql_replica_database_engine_1/test.py::test_concurrent_transactions test_postgresql_replica_database_engine_1/test.py::test_different_data_types test_postgresql_replica_database_engine_1/test.py::test_drop_database_while_replication_startup_not_finished test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_all_database_tables test_postgresql_replica_database_engine_1/test.py::test_load_and_sync_subset_of_database_tables test_postgresql_replica_database_engine_1/test.py::test_many_concurrent_queries test_postgresql_replica_database_engine_1/test.py::test_multiple_databases test_postgresql_replica_database_engine_1/test.py::test_quoting_1 test_postgresql_replica_database_engine_1/test.py::test_quoting_2 test_postgresql_replica_database_engine_1/test.py::test_replica_identity_index test_postgresql_replica_database_engine_1/test.py::test_replicating_dml test_postgresql_replica_database_engine_1/test.py::test_restart_server_while_replication_startup_not_finished test_postgresql_replica_database_engine_1/test.py::test_single_transaction test_postgresql_replica_database_engine_1/test.py::test_table_schema_changes test_postgresql_replica_database_engine_1/test.py::test_user_managed_slots test_postgresql_replica_database_engine_1/test.py::test_virtual_columns test_quota/test.py::test_add_remove_interval test_quota/test.py::test_add_remove_quota test_quota/test.py::test_consumption_of_show_clusters -vvv" altinityinfra/integration-tests-runner:cf95077f4609 ' returned non-zero exit status 1.